Interactive order fulfillment system

ABSTRACT

Embodiments disclose an interactive order fulfillment system, as well as a corresponding method and computer-readable storage medium, in which indications of order fulfillment status are displayed onto a surface of a receptacle used for fulfilling the order. The interactive order fulfillment system may operate as a virtual receipt with which an employee may interact to more efficiently fulfill the order.

BACKGROUND

This disclosure relates generally to order fulfillment systems, and moreparticularly to an interactive order fulfillment system using orderreceptacles as interactive surfaces.

Many industries rely on accurate and efficient order fulfillment intheir day-to-day operations, the industries ranging anywhere from smallretail and food service establishments to massive supply and logisticscenters having global distribution networks. Especially in thoseindustries that are not highly automated, human employees are heavily,often entirely, relied upon to fulfill orders. Idiosyncrasies ofindividual employees, when combined with other human factors, such asboredom, stress, inattention, forgetfulness, etc. often lead to mistakesor inefficiency in the order fulfillment process.

SUMMARY

One embodiment provides a method that includes displaying an indicationof an order fulfillment status onto a surface of a receptacle. Thereceptacle is generally configured to receive one or more of the orderitems. The method further includes updating the order fulfillment statusbased on a detected interaction with the displayed indication.

Another embodiment provides an interactive order fulfillment system thatincludes a receptacle for receiving one or more items from the order, adisplay device configured to display an indication of an orderfulfillment status onto a surface of the receptacle. The interactiveorder fulfillment system further includes an input device configured todetect interaction with the displayed indication, and a controllerconfigured to update the order fulfillment status based on the detectedinteraction.

Yet another embodiment provides a computer-readable storage mediumstoring an application. When executed on a processor, the applicationperforms an operation that includes displaying an indication of an orderfulfillment status onto a surface of a receptacle, where the receptacleconfigured to receive one or more items included in the order. Theoperation further includes updating the order fulfillment status basedon a detected interaction with the displayed indication.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 depicts an interactive order fulfillment system, according to oneembodiment described herein.

FIGS. 2A-2C depict an interactive surface in an interactive orderfulfillment system, according to one embodiment described herein.

FIGS. 3A and 3B depict an interactive surface in an interactive orderfulfillment system, according to one embodiment described herein.

FIGS. 4A and 4B depict an interactive surface in an interactive orderfulfillment system, according to one embodiment described herein.

FIG. 5 depicts an example of an interactive order fulfillment system,according to one embodiment described herein.

FIG. 6 is a block diagram depicting a method for order fulfillment,according to one embodiment described herein.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures. It is contemplated that elements disclosed in oneembodiment may be beneficially utilized on other embodiments withoutspecific recitation. The drawings referred to here should not beunderstood as being drawn to scale unless specifically noted. Also, thedrawings are often simplified and details or components omitted forclarity of presentation and explanation. The drawings and discussionserve to explain principles discussed below, where like designationsdenote like elements.

DETAILED DESCRIPTION

Described in various embodiments of this disclosure is an interactiveorder fulfillment system, in which indications of order fulfillmentstatus may be displayed onto a surface of a receptacle used forfulfilling the order; that is, the receptacle is used for holding one ormore items from the order as the order is being fulfilled. Withincertain industries, such as retail shopping or food service, theinteractive order fulfillment system may operate as a virtual receiptwith which an employee may interact to more efficiently fulfill theorder.

The order fulfillment status may generally be represented using anyfeasible content, formatting, and layout; each of these may be selectedto improve the presentability or readability of the displayedindication, to improve the efficiency of corresponding interactions withthe displayed indication, and/or according to any number of additionalcriteria. The indications may include a partial or complete textuallisting or graphical display of items included in the order, or asimilar listing or display that selectively includes certain items orgroups of items (such as order items that have not yet been fulfilled).

The interactive order fulfillment system may be networked with numerousother computing systems related to the order fulfillment process, suchas systems controlling or monitoring inventory, item assembly orproduction, transportation, etc. Alternatively, the interactive orderfulfillment system may natively include such functionality. By usingdata generated during the order fulfillment process (often provided bythe networked systems), the interactive order fulfillment system mayprovide more effective indications of order fulfillment status. Theindications may include status(es) of order items, estimate(s) of timeremaining until fulfillment of particular items (or of the entireorder), etc.

As mentioned above, indications of order fulfillment status maygenerally be presented in textual or graphical formats, or may includecombinations thereof. For example, the indications may includecolor-coded indications reflecting a status of the collective order, orreflecting statuses of certain items or groupings of items. The colorcoding schemes may include colors reflecting discrete levels offulfillment (such as green representing fulfilled items, yellow forpartially fulfilled, and red for not fulfilled), or may includecontinuous color gradients reflecting some measure of fulfillment (suchas a hot/cold gradient based on an estimated value, such as timeremaining). Of course, differing graphical schemes may be selected foruse at different levels. For example, individual items may berepresented by discrete-level color indicators, and groups of thoseitems may use a single color indicator corresponding to an average valueof the component items' color indicators, or may display each of thecomponent items' color indicators in a matrix, or using any othersuitable graphical display techniques.

FIG. 1 depicts an interactive order fulfillment system, according to oneembodiment described herein. Interactive order fulfillment system 100includes one or more receptacles 110, a display device 130, a controller140, and one or more input devices 150. Receptacle 110 (shown here in atop-down perspective) includes at least one surface, such as a bottomsurface 112 or one or more walls 111. Some non-limiting examples ofreceptacle 110 include boxes, containers, trays, buckets, crates, or anyother device suitable for containing and/or transporting other items. Ofcourse, depending on size and configuration, any surface of receptacle110 may be suitable for displaying order fulfillment status consistentwith descriptions provided herein. For simplicity, receptacle 110 isshown as containing one or more order items 120; of course, receptacle110 need not always contain order items (i.e., the receptacle may beempty), and may further contain other, non-order items. Each of theorder items 120 may vary in shape, size, weight, etc.

Controller 140 includes one or more processors 142, memory 144, andinput/output (I/O) 146. Processor 142 is generally configured to readand execute code stored in memory 144, and to communicate with variousperipheral devices and/or other computing systems through I/O 146.Memory 144 may include both volatile memory, such as various types ofrandom access memory (RAM), as well as one or more forms of non-volatilememory (e.g., hard drives, solid state drives, Flash memory, etc.).Importantly, controller 140 may maintain data related to orderfulfillment status within memory 144. As shown, the controller I/O 146is connected through network 125 to display device 130, input devices150, and computing systems 160. Network 125 may represent multiple typesof physical and logical connections; for example, display device 130 andinput devices 150 may be directly coupled to controller 140 via videoand other cabling (as in an ordinary personal computer configuration),while controller 140 is communicatively coupled to other computingsystems 160 through a local access network (LAN) or wide area network(WAN), whether through a wired or wireless connection.

The display device 130 may include any device capable of displaying ontoone or more surfaces of a receptacle in a manner consistent with thedescriptions provided herein. Display device 130 may be configured todisplay in display area 115 onto at least a portion of the surface. Thedisplay area may have a fixed size and position, but as will bedescribed more fully below, certain embodiments may allow the displayarea to be dynamically moved on the surface, or even onto differentsurfaces. Within display area 115, the display device 130 may alsodisplay one or more order fulfillment status indicators 117, as aredescribed briefly above and will be further described below. In oneembodiment, the display device may be a projector configured to projectonto a surface of receptacle 110. Another embodiment may provide displaydevice 130 as a liquid crystal display (LCD) or other display technologyintegrally formed in the receptacle, such as an LCD embedded in asurface of a reusable tray.

Input devices 150 may include one or more different types of sensors ordevices capable of detecting input at or proximate to receptaclesurfaces. Input devices 150 may include one or more of weight sensors152, touch sensing devices 154, temperature sensors 156, and visualrecognition devices 158. Of course, other input devices may bealternately or additionally employed, such as sensors or devicesconfigured to read coding on order items (e.g., serial barcodes or QuickResponse (QR) codes), sensors or devices configured to electricallycouple with elements of the order items (e.g., radio-frequencyidentification (RFID) tags included with the order items and readersincluded in or near receptacles), and so forth.

Weight sensors 152 may include strain gages or piezoelectric elementsdisposed within or adjacent to the receptacle 110 and that are capableof measuring weight of the one or more order items 120 as they areplaced into the receptacle. Multiple weight sensors 152 may be disposedin an arrangement along a particular receptacle surface in order tosense weight distribution with a greater resolution, which in turn mayimprove the identification of order items 120.

Touch sensing devices 154 may include touch sensors, such as resistive,inductive or capacitive touch sensors (or using any other touch sensingtechnology); in certain embodiments, the touch sensing devices may beintegrally formed in a surface of the receptacle. The touch sensingdevices are generally selected to be compatible with the display device130; for example, the LCD disclosed above may include a transparentsurface with an array of touch sensors, or the array of touch sensorsmay be included in a surface of the receptacle while a projectordisplays onto the surface (or another surface).

In an alternate embodiment, touch sensing devices 154 may include adepth camera used to determine relative distances between detectedobjects and a surface of the receptacle. By using a depth camera, anordinary surface may be interactively used by an individual. Indetermining the relative distances, the system may be able to positivelyidentify the detected objects. The detected objects may include the oneor more order items 120, and may also include manual input from a user(i.e., touch with one or more fingers). Calibration and the distancedeterminations may be performed by computing logic included with thedepth camera, or may be performed by processor 142 as data is receivedat controller 140.

Temperature sensors 156 may also be included for item identification,and may further provide quality control for the order fulfillment. Forexample, certain order items may be heated to different temperaturesbefore order fulfillment (e.g., food service) or may be stored atdifferent temperatures. Additionally, to ensure customer satisfaction,the temperature sensors may be used during the order fulfillment process(for example, just prior to completion of the order fulfillment, atwhich point a customer may receive their food). The temperature sensors156 may be able to identify items by an absolute or relative temperaturevalue, or a spatial temperature distribution (which is related to itemsize, another identifying feature for items). The temperature sensors156 may include thermocouples, thermistors, or resistance temperaturedetectors (RTDs), which are preferably located proximate to thereceptacle to more accurately detect temperature of any items containedtherein. Alternately, the temperature sensors 156 may include aninfrared (IR) sensor configured to detect heat emitted by itemscontained within the receptacle; this may include IR sensors proximateto the receptacle, or perhaps an IR camera at some distance from thereceptacle. In one embodiment, an IR camera may be disposed adjacent toa projector that displays onto a receptacle surface, and the IR cameraand projector may be included in a common housing.

Visual recognition devices 158 may also be included for itemidentification, according to any feasible techniques familiar to aperson of ordinary skill in the art. In one non-limiting example, anordinary camera or charge-coupled device (CCD) sensor may be included tocapture images of the surface, and the processor 142 may perform imageprocessing on the captured images to determine whether one or more itemshave been placed onto the surface, and to identify the placed items. Inone embodiment, the processor 142 may perform visual recognition usingartificial neural networks implemented in hardware, or included insoftware stored on memory 144.

Controller 140 may also be connected through network 125 to one or morecomputing systems 160, which may perform certain ancillary functionsrelated to order fulfillment such as inventory management 162, itemassembly, production, or preparation 164, financial functions 166, etc.Alternately, some or all of these functions may be performed bycontroller 140. Output data from various functions of the computingsystems 160 may be provided to controller 140, which using processor 142may perform computation or other processing on the data, and may updateany order fulfillment status calculations (such as an estimated time tofulfillment for various items).

FIG. 2A depicts an interactive surface in an interactive orderfulfillment system, according to one embodiment described herein. Theinteractive surface 112 may generally be included as part of fulfillmentsystem 100. Display area 115 is included within the area of surface 112,and within display area 115 are one or more status indicators 117. Forsimplicity of description, the indicators 117 are depicted here as atextual listing of order items 205 ₁-205 _(N), and at an initial sizeand position within display area 115; of course, the indicators 117 maybe displayed in alternate textual or graphical formats (or incombinations thereof) generally as described above.

Beyond merely indicating order fulfillment status, in certainembodiments the interactive order fulfillment system may also provide aprescribed placement within the receptacle (or between differentreceptacles) for those order items that have not yet been fulfilled. Theinteractive order fulfillment system may display placement locations forthese unfulfilled items onto the receptacle surface using text or simplegraphics; for example, a text box containing the name of the particularitem. In certain embodiments, the system may display detailed (e.g.,photographic) representations of the items onto the receptacle surface,so that the surface appears similar to how it will appear when the orderis entirely fulfilled according to the prescribed placement. Providingthese detailed representations may increase accuracy and efficiency ofthe order fulfillment, especially where one or more individuals aremanually fulfilling the order. The visual representations may be quicklyinterpreted and understood by the individual(s) fulfilling the order,even in a stressful or fast-paced environment, and may reduce any needfor consideration of item placement or for communication andcoordination between individual(s). Further, any incorrect placement ofan item may be quickly identified and corrected. For example, the systemmay display photos of a book having a red cover onto a portion of a boxsurface, and another book having a black cover onto another portion ofthe surface; using the displayed photos and their placement, twoemployees may simultaneously work to fulfill the order (say, eachemployee retrieves one book and places their respective book onto thematching display portion) without unnecessary consideration orcoordination.

The interactive order fulfillment system may determine its prescribedplacements based on any number of different factors, such as maximizingthe available area or volume of the receptacle, meeting certain weightthresholds (e.g., minimum and/or maximum weights) for items in thereceptacle, optimizing a weight distribution of items within thereceptacle, protecting less durable items, etc. For example, for a fastfood order including a hamburger, French fries, and soft drink, thesystem may prescribe that the (presumably) heavier soft drink be placedtowards the center of the tray, thereby reducing instability as the trayis carried by a customer. The system may also distribute order items tobe fulfilled using multiple receptacles based on size or weight of theitems, as well as the size or weight capacity of the receptacle; ofcourse, this may include distribute items across different types ofreceptacles (i.e., having different sizes or strength ratings). Forexample, certain order items (or combinations of items) might be tooheavy or dense so that the items are not suitable to be placed in astandard cardboard packaging box, but could instead be placed in aplastic crate having greater strength.

As order items are fulfilled, the interactive order fulfillment systemmay update its display accordingly. For example, order items which havealready been correctly fulfilled (i.e., correct items and placedaccording to prescribed placement locations) may no longer require thelocation to be displayed at the receptacle surface. Further, any itemsthat are incorrectly fulfilled (e.g., an incorrect item, an incorrectplacement, an incorrect weight for the item, etc.) may affect thefactors considered in determining the prescribed placements, so that thesystem may be required to update placement locations for the remainingunfulfilled items.

In response to any type of incorrect fulfillment, the system may providean alert using the display, and may maintain the alert until the erroris acknowledged or corrected. Alternatively, in the case of a correctitem that was incorrectly placed (or that perhaps shifted afterplacement), the system may adaptively update its placement locations,determining new placement locations for one or more unfulfilled itemsbased on the properties of the item incorrectly placed.

In some embodiments, the system may weigh items as each is placed in thereceptacle, providing an alert indicating that the item's determinedweight does not equal (or falls outside some range of) an expectedweight for the item. The expected weight values (against which thedetermined weights are compared) generally may be stored in a memory ofthe interactive order fulfillment system, or accessed from a networkedcomputing system. In other embodiments, items are selectively weighedaccording to any number of criteria; one criterion for weighing itemsmay be whether additional employee actions are necessary for the item'sfulfillment. For example, a soft drink cup may have an expected weightthat is based on the volume of liquid added, which is expected to beadded by an employee. Weight checking may thus provide an additionalquality control measure to the fulfillment process.

FIGS. 2B and 2C depict an interactive surface in an interactive orderfulfillment system, according to one embodiment described herein. Theinteractive surface 112 may generally be included as part of fulfillmentsystem 100. Beyond including status indicators 117, the display area 115includes an item placement area 225. While shown as one contiguous areafor simplicity, the item placement area 225 may be divided into multipledistinct areas and dispersed across the display area 115. Within itemplacement area 225, the prescribed placements 235 ₁-235 ₆ are disposed.In this example, the order items 205 included in the status indicator117 (order items 205 ₁-205 _(N), as shown in FIG. 2A) each correspond toa different prescribed placement. Assume for purposes of this examplethat order item 205 ₁ corresponds to prescribed placement 235 ₁, 205 ₂to 235 ₂, and so forth.

The prescribed placements 235 may be determined according to any of thedifferent factors described above. For example, assume that interactivesurface 112 is a tray having relatively short side walls and that itemsplaced onto the tray may fall off (or tip over, etc.) during subsequentmovement of the tray. The interactive order fulfillment system 100,using its controller 140, may determine (based on properties of orderitems to be fulfilled, and properties of the tray itself) that the totalweight of items 205 ₁-205 ₆ corresponding to placements 235 ₁-235 ₆ iswithin a maximum weight threshold for the tray (i.e., the tray will beable to support the total weight), and that heavier items should beplaced nearer to the center of the surface in order to preventunnecessary imbalance of the tray. The interactive order fulfillmentsystem 100 may also include preferences for display of the statusindicators 117, such preferences being included either in the softwarein memory 144, or received as an input to the controller 140. Assumehere that a preference has been specified for display in one contiguousarea and adjacent to item placement area 225. The controller 140 may thedetermine the layout of prescribed placements 235 and status indicators117 that are depicted in FIG. 2B, based on the weights of the orderitems to be fulfilled, the properties of the receptacle (i.e., thetray), and the display preferences.

In FIG. 2C, and continuing the example, a first order item 205 ₁ hasbeen placed onto the interactive surface 112. However, instead of beingplaced at the location of prescribed placement 235 ₁, item 205 ₁ wasplaced at a location that overlapped the displayed status indicator(former position 217). The interactive order fulfillment system 100 insome embodiments may be configured to produce an alert in response tosuch an incorrect fulfillment; however, for purposes of this example,assume that the interactive order fulfillment system 100 is configuredto adaptively update placement of the remaining unfulfilled items basedon the incorrectly placed item (i.e., its weight and location).Considering also the earlier specified preference for one contiguousdisplay area for status indicators 117, the controller 140 may thendetermine the layout of prescribed placements 235 and status indicators117 as shown in FIG. 2C, and may move these display elementsaccordingly. This item fulfillment process may generally repeat asadditional items are placed onto the interactive surface 112, endingwhen the order has been fulfilled (i.e., when all order items 205included on the interactive surface 112). The item fulfillment processmay also end upon certain events, such as determining an inability tomeet all the parameters for prescribed placements. For example, theincorrect placement of order item 205 ₁ may have created a condition inwhich determining an acceptable weight distribution (i.e., withinreceptacle ratings and/or an estimate of imbalance) was impossible. Ofcourse, this is meant to be only one non-limiting example; otherconfigurations consistent with the above description are possible, suchas distributing order items and displaying status indicators acrossmultiple receptacles, identifying and selecting receptacles of differenttypes for order fulfillment, overriding preferences, weighteddeterminations, and so forth.

The interactive order fulfillment system may update the orderfulfillment status as order items are fulfilled, and may in turn displaythe updated order fulfillment status. Typically, order items arefulfilled by placing the items into the receptacle. In some embodiments,the interactive order fulfillment system may automatically detect whenitems have been placed into the receptacle, such as by using weightmeasurements, visual recognition, temperature characteristics, etc. Asdescribed above, various input devices 150 may be used to accomplishdetection of the order items, and may include weight sensors, touchsensing devices, temperature sensors, visual recognition devices, aswell as other devices known in the art for performing detection.Additionally or alternatively, an individual fulfilling the order maymanually indicate that certain items have been fulfilled, for example byentering input through a user interface. The interactive orderfulfillment system may also receive user inputs through touch sensingdevices, such as discrete touch sensors or with a depth cameraconfiguration, essentially as described above.

FIGS. 3A and 3B depict a display area in an interactive orderfulfillment system, according to one embodiment described herein. Thedisplay area 115 may generally be included on an interactive surface 112as a part of fulfillment system 100. In this embodiment, statusindicators 117 include a textual listing of order items by item number305, a fulfillment status 315, and a time remaining 325. Of course, theorder fulfillment status may be displayed using different data or in anyalternate textual or graphical manner, consistent with the descriptionsprovided above. For example, the status indicators may include afulfillment status or time remaining for the entire order, in additionto the item-level data provided in FIG. 3A. As shown, the fulfillmentstatus 315 for each item is color-coded based on the estimate of timeremaining 325. Using the red-yellow-green color scheme describedearlier, fulfilled items correspond to a fulfillment status value ofgreen ([GRN]), items with remaining times equal to or less than oneminute and 30 seconds (1:30) correspond to a fulfillment status value ofyellow ([YEL]), and for remaining times exceeding 1:30, the fulfillmentstatus value is red ([RED]). Of course, any color scheme or varioustimes or other criteria may be selected to indicate fulfillment status.

In FIG. 3B, and continuing the example, a first order item 205 ₁ hasbeen placed onto the interactive surface 112 at its prescribed placement235 ₁. Depending on the input devices used for item detection in theparticular embodiment, interactive order fulfillment system mayautomatically detect the (correct) placement of the order item 205 ₁ andautomatically update the status indicator using controller 140. Forexample, the fulfillment status 315 corresponding to the item may bechanged to [GRN], and the time remaining 325 may be deleted or a zerovalue; in some embodiments, the listing corresponding to the fulfilleditem 205 ₁ may be removed entirely from the status indicator.Alternatively, a user may manually interact with the interactive surface(perhaps by touching the surface at the location 335 the fulfillmentstatus field is displayed), and select the updated [GRN] color orotherwise indicate that the item has been fulfilled.

In some embodiments, the interactive order fulfillment system maysupport simultaneous, separate interactions on the receptacle surface.In certain contexts (e.g., fast food or retail service), the receptaclemay be located on a counter between the employee(s) fulfilling theorder, and the customer awaiting receipt of the order. While the orderis being fulfilled, the customer may interact with a separate displayinterface on the receptacle, whether for entertainment (e.g., displayinginteractive games or videos) or to modify their placed order. Forexample, a fast food customer while waiting may realize that she forgotto order extra pickles on her hamburger, and perhaps decides to changethe size of the soft drink in her order. The customer, by interactingwith the separate display interface during order fulfillment, may add orremove items, or modify items from the placed order, and if necessarypay any difference in price. One or more display devices may be used toproject the separate display interface onto the surface. In oneembodiment, both the order fulfillment interface and the customerinterface are displayed using one display device, which may orient eachinterface to its intended user (e.g., the order fulfillment interfaceoriented for viewing by employee(s) fulfilling the order, and thecustomer interface oriented for viewing by the customer.

FIGS. 4A and 4B depict a display area in an interactive orderfulfillment system, according to one embodiment described herein. Thedisplay area 115 may generally be included on an interactive surface 112as a part of fulfillment system 100. An order fulfillment interface 405and a customer interface 415 are shown. The order fulfillment interface405 may include status indicators 117 and prescribed placements 235, andmay operate the same or substantially similarly to the embodimentsdepicted in FIGS. 1-3B and as described above. Each interface may bedisplayed onto the interactive surface (or alternately, onto separatesurfaces) using separate display devices 130, or each may be displayedusing a single display device 130. The interfaces may be displayed withdifferent orientations, such as with a 180° shift. While each interfaceis shown as being contiguous, and as being comparable in size andadjacent to one another on the interactive surface, the interfaces 405,415 may vary in size, distribution, relative positioning, and so forth.

FIG. 4B further depicts a simultaneous interaction with the interfaces405, 415 during order fulfillment. As shown, several order items 205₁-205 ₄ have been placed onto the surface of the receptacle, and theitems corresponding to prescribed placements 235 _(N-1) and 235 _(N)have yet to be fulfilled. Though perhaps not immediately apparent, theorder fulfillment interface 405 is oriented at an employee 455 and at anapproximately 180° shift from customer interface 415, so that the statusindicators 117 may be easily read and understood by located on one sideof the receptacle and who may be fulfilling the order. Customerinterface 415 is oriented toward a customer 465 and may include, forexample, a menu 425 of additional items for addition and/or purchase, aswell as a payment menu 435. In some embodiments, the payment menu 435may be configured to receive the customer's payment while allowingemployees to continue fulfilling the order, that is, without requiringthe employee to complete another cash register transaction. Accordingly,payment selections in the payment menu 435 may include using a previouspayment method, such as a credit or debit card or electronic accountthat was used to complete the initial order transaction, or using a newpayment method (e.g., a different card). Other embodiments may providemenu options for alternate forms of payment (such as cash, checks,etc.). After payment, changes to the order made by the customer 465 maybe reflected in the status indicators 117, such as by adding entries fornewly ordered items, etc.

FIG. 5 depicts an example of an interactive order fulfillment system,according to one embodiment described herein. A retail counter 500generally separates employee-only areas from those accessible bycustomers. The counter 500 includes a cash register 525 and a near fieldcommunication (NFC) reader 535, which may be coupled to the cashregister 525 and is typically used to receive contactless payments. Thecash register 525 and NFC reader 535 are examples of computing system(s)160 networked to the interactive order fulfillment system 100. Aftercompleting an order transaction, a tray 545 (an example of receptacle110) may be placed onto the counter 500 and the order fulfillment statusdisplayed onto a surface 550. A projector 505 (an example of displaydevice 130) may display the order fulfillment interface 405, as well asa separate customer interface 415, onto a display area 115 comprising atleast part of the surface 550. The interfaces may be orienteddifferently and towards their intended users; order fulfillmentinterface 405 may be oriented to employee 455, who is on one side ofcounter 500, and customer interface 415 towards customer 465 on theother side of the counter 500. A depth camera 515 (example of touchsensing device 154) may be configured to detect input at or near thedisplay area 115, such as touch input from customer 465 or employee 455,or from fulfilled order items 205 placed into the tray 545. During theorder fulfillment process, customer 465 may select additional items forpurchase, which updates the order and the order fulfillment statusindicators accordingly.

FIG. 6 is a block diagram depicting a method for order fulfillment,according to one embodiment described herein. The method may generallybe used by elements of interactive order fulfillment system 100, and mayinclude features of the various embodiments disclosed in FIGS. 1-5 anddescribed above.

The method begins at block 600, where an order transaction is completed.Consistent with examples above, completing the transaction may includeselection and payment for items of an initial order. This initial ordermay later be modified during the fulfillment process, as items areadded, removed, or altered by a customer, or are fulfilled byemployee(s).

At block 610, an indication of order fulfillment status is displayedonto a surface of a receptacle. This may include textual and/orgraphical status indicators for items included in the order, groups ofitems, and/or for the entire order itself.

At block 620, the interactive order fulfillment system determineswhether an interaction with the displayed indication has been detectedby one or more input devices. This may include manual (i.e., touch)input from a customer or employee, or may be in response to detectingand identifying order items placed into the receptacle.

If an interaction is detected, the method proceeds to block 625, wherethe interactive order fulfillment system 100 updates the orderfulfillment status(es) based on the detected interaction. This mayinclude updates as items from the order are fulfilled, as well as if acustomer modifies the initial order during the fulfillment process. Themethod returns to block 610, where the updated order fulfillment statusis displayed.

However, if an interaction is not detected at block 620, the methodinstead proceeds to block 630, where the interactive order fulfillmentsystem determines whether the order has been completely fulfilled. Ifthe order has remaining unfulfilled items, the method may continue toloop between blocks 620 and 630 until an interaction is detected.Alternately, block 620 may return to itself if no interaction isdetected, acting essentially as a wait state. If the order has beencompletely fulfilled at block 630, the method may end.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Embodiments of the invention may be provided to end users through acloud computing infrastructure. Cloud computing generally refers to theprovision of scalable computing resources as a service over a network.More formally, cloud computing may be defined as a computing capabilitythat provides an abstraction between the computing resource and itsunderlying technical architecture (e.g., servers, storage, networks),enabling convenient, on-demand network access to a shared pool ofconfigurable computing resources that can be rapidly provisioned andreleased with minimal management effort or service provider interaction.Thus, cloud computing allows a user to access virtual computingresources (e.g., storage, data, applications, and even completevirtualized computing systems) in “the cloud,” without regard for theunderlying physical systems (or locations of those systems) used toprovide the computing resources.

Typically, cloud computing resources are provided to a user on apay-per-use basis, where users are charged only for the computingresources actually used (e.g. an amount of storage space consumed by auser or a number of virtualized systems instantiated by the user). Auser can access any of the resources that reside in the cloud at anytime, and from anywhere across the Internet. In context of the presentinvention, the interactive order fulfillment system may accessapplications or separate computing systems related to the orderfulfillment process (e.g., inventory management, item assembly orpreparation, transportation, etc.) or related data available in thecloud. For example, an inventory management application could execute ona computing system in the cloud and provide inventory-related data tothe interactive order fulfillment system in order to generate moreaccurate estimates of order fulfillment status and projected completion.In such a case, the inventory management application could monitor stockquantities at several dispersed warehousing locations and store the dataat a storage location in the cloud. Doing so generally allows anycomputing system that is attached to a network connected to the cloud(e.g., the Internet) to access this data.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

What is claimed is:
 1. A method, comprising: displaying an indication ofan order fulfillment status onto a surface of a receptacle, thereceptacle configured to receive one or more items included in theorder; and based on a detected interaction with the displayedindication, updating the order fulfillment status.
 2. The method ofclaim 1, wherein the indication is displayed onto the surface using aprojector.
 3. The method of claim 1, wherein the detected interactioncomprises one or more of detecting an item placed into the receptacle,and manually indicating a change to the order fulfillment status.
 4. Themethod of claim 1, wherein displaying the indication further comprisesindicating one or more locations within the receptacle for placing acorresponding one or more unfulfilled items from the order.
 5. Themethod of claim 4, further comprising adaptively updating the indicatedone or more locations, based on the detected interaction.
 6. The methodof claim 1, wherein the interaction is detected using one or more of acamera configured to distinguish items placed into the receptacle, adepth camera configured to detect touches at the surface of thereceptacle, and a weighing device configured to weigh the receptacle andone or more items placed therein.
 7. The method of claim 1, wherein theindication includes one or more of a list including at least oneunfulfilled item from the order, a color-coded indication of the orderfulfillment status, and an estimated time until fulfillment of theorder.
 8. The method of claim 1, wherein displaying the indication ontothe surface occurs with a first orientation, and further comprisingdisplaying a customer interface onto the surface with a secondorientation.
 9. The method of claim 8, further comprising updating theorder based on an interaction with the customer interface.
 10. Aninteractive order fulfillment system, comprising: a receptacle forreceiving one or more items from the order; a display device configuredto display an indication of an order fulfillment status onto a surfaceof the receptacle; an input device configured to detect interaction withthe displayed indication; and a controller configured to, based on thedetected interaction, update the order fulfillment status.
 11. Theinteractive order fulfillment system of claim 10, wherein the displaydevice comprises a projector.
 12. The interactive order fulfillmentsystem of claim 10, wherein the detected interaction comprises one ormore of detecting an item placed into the receptacle, and manuallyindicating a change to the order fulfillment status.
 13. The interactiveorder fulfillment system of claim 10, wherein the display device isfurther configured to indicate one or more locations within thereceptacle for placing a corresponding one or more unfulfilled itemsfrom the order.
 14. The interactive order fulfillment system of claim13, wherein the display device is further configured to adaptivelyupdate the indicated one or more locations, based on the detectedinteraction.
 15. The interactive order fulfillment system of claim 10,wherein the input device comprises one or more of a camera configured todistinguish items placed in the receptacle, a depth camera configured todetect touches at the surface of the receptacle, and a weighing deviceconfigured to weigh the receptacle and one or more items placed therein.16. The interactive order fulfillment system of claim 10, wherein theindication includes one or more of a list including at least oneunfulfilled item from the order, a color-coded indication of the orderfulfillment status, and an estimated time until fulfillment of theorder.
 17. The interactive order fulfillment system of claim 10, whereinthe display device is configured to display the indication onto thesurface with a first orientation, and is further configured to display acustomer interface onto the surface with a second orientation.
 18. Theinteractive order fulfillment system of claim 17, wherein the controlleris further configured to update the order based on an interaction withthe customer interface.
 19. A computer-readable storage medium storingan application, which, when executed on a processor, performs anoperation, comprising: displaying an indication of an order fulfillmentstatus onto a surface of a receptacle, the receptacle configured toreceive one or more items included in the order; and based on a detectedinteraction with the displayed indication, updating the orderfulfillment status.
 20. The computer-readable storage medium of claim19, wherein displaying the indication onto the surface occurs with afirst orientation, and further comprising displaying a customerinterface onto the surface with a second orientation.